Supplementary Materials for “Distributed Newton Methods for Deep Neural Networks”

Notation Description y The label vector of the ith training instance. x The feature vector of the ith training instance. l The number of training instances. K The number of classes. θ The model vector (weights and biases) of the neural network. ξ The loss function. ξi The training loss of the ith instance. f The objective function. C The regularization parameter. L The number of layers of the neural network. nm The number of neurons in the mth layer. n0 The number of input neurons (the dimension of the feature vector). nL The number of output neurons (the number of classes, except for binary classification one may use nL = 1). W The weight matrix in the mth layer (with dimension <nm−1×nm). w tj The weight between neuron t in the (m − 1)th layer and neuron j in the mth layer. w The vector obtained by concatenating the columns of W. b The bias vector in the mth layer. s The affine function (Wm)Tzm−1,i+b in themth layer for the ith instance. z The output vector (element-wise application of the activation function on s) in the mth layer for the ith instance. σ The activation function. n The total number of weights and biases. J i The Jacobian matrix of z with respect to θ. J i p The local component of the J i in the partition p.